storage: override Pebble's ExitFunc to route through fatal logging#166840
storage: override Pebble's ExitFunc to route through fatal logging#166840williamchoe3 wants to merge 1 commit into
Conversation
Set pebble.Options.ExitFunc so that Pebble invariant violation exits (cache assertions, memtable checks, sstable reader checks) route through CockroachDB's log.Fatal path instead of calling os.Exit(1) directly. This gives visibility into why the process died via log lines, Sentry crash reports, and fatal exit marker files. Requires cockroachdb/pebble#5865. Co-Authored-By: roachdev-claude <roachdev-claude-bot@cockroachlabs.com>
|
Merging to
|
|
It looks like your PR touches production code but doesn't add or edit any test code. Did you consider adding tests to your PR? 🦉 Hoot! I am a Blathers, a bot for CockroachDB. My owner is dev-inf. |
|
Alternative approach: skip The current implementation routes Pebble's A lighter approach that minimizes the window of running with corrupted state: cfg.opts.ExitFunc = func(code int) {
msg := fmt.Sprintf("pebble: invariant violation (exit code %d)", code)
// Write marker file directly — no mutex, no logging infrastructure
for _, spec := range stores {
auxDir := filepath.Join(spec.Path, "auxiliary")
_ = os.WriteFile(filepath.Join(auxDir, "_FATAL_EXIT.txt"), []byte(msg), 0644)
}
// Exit immediately — don't go through log.Fatal
os.Exit(code)
}Tradeoffs:
The direct approach is safer for invariant violations where in-process memory may be corrupted (e.g. cache linked list pointers, reference count underflow). The |
Summary
Set
pebble.Options.ExitFuncso that Pebble invariant violation exits (cache assertions, memtable checks, sstable reader checks) route through CockroachDB'slog.Fatalpath instead of callingos.Exit(1)directly.Without this, Pebble's 18 direct
os.Exit(1)calls bypass all logging, crash reporting (Sentry), and marker file writing — the process just vanishes with exit code 1 and no indication of why.Depends on: cockroachdb/pebble#5865 (adds
Options.ExitFuncto Pebble)Note
This PR will not compile until the Pebble dependency is updated to include cockroachdb/pebble#5865.